Why emotion matters

Although much is known about the representation and processing of concrete concepts, our knowledge of what abstract semantics might be is severely limited. In this paper we first address the adequacy of the two dominant accounts (dual coding theory and the context availability model) put forward in order to explain representation and processing differences between concrete and abstract words. We find that neither proposal can account for experimental findings and that this is, at least partly, because abstract words are considered to be unrelated to experiential information in both of these accounts. We then address one particular type of experiential information, emotional content, and demonstrate that it plays a crucial role in the processing and representation of abstract concepts: statistically, abstract words are more emotionally valenced than concrete words and this accounts for a residual latency advantage for abstract words, when variables such as imageability (a construct derived from dual coding theory) and rated context availability are held constant. We conclude with a discussion of our novel hypothesis for embodied abstract semantics. The representation of abstract word meanings: Why emotion matters Concrete entities exist in space-time and are independent of human minds/language; abstract entities, on the other hand, do not exist in space-time but their existence depends on human minds/language (Hale, 1988). “Concreteness”, therefore, indexes a basic ontological distinction, dividing entities into these two kinds. This ontological distinction is reflected in our epistemologies, and concreteness is arguably an organizing principle of semantic knowledge. Up to the present, research into semantic and conceptual representation has focused almost exclusively on how concrete word meanings and Embodied abstract semantics 3 concepts are represented and processed, to the exclusion of abstract meanings and concepts. However, the ability to communicate through language about abstract concepts, such as courage, dignity, revenge, lies at the heart of what it means to be human, and no theory of semantic or conceptual representation is complete without an explicit account of how abstract knowledge is acquired, represented, and processed. In this paper we first demonstrate, combining experiments with large scale regression analyses of data from the English Lexicon Project (ELP, Balota et al., 2007), that the dual coding theory and the context availability hypothesis two of the most popular accounts of differences in representation and processing between concrete and abstract words do not exhaustively account for processing (and hence representational) differences between the two types of word meanings. In fact, once imageability and context availability (along with a large number of other lexical and sublexical variables) are controlled, there is a residual advantage for abstract word processing. We show that this advantage can be explained by differences in emotional valence between concrete and abstract words, and we discuss a new hypothesis of how the semantic system is organized with respect to the distinction between concrete and abstract concepts. Specifically, we propose that both concrete and abstract concepts bind different types of information: experiential information (sensory, motor and affective), and also linguistic information. However, concrete and abstract semantic representations differ in terms of whether sensory, motor or affective information have the greatest weight, with sensory-motor information being more preponderant for concrete concepts and affective information playing a greater role for abstract concepts. Thus, a central and novel element of this proposal is the idea that experiential information contributes to the representation of both concrete and abstract words, however, whereas sensory-motor information is statistically more important for the Embodied abstract semantics 4 representation of concrete words, emotional content, a largely neglected type of experiential information in the literature on semantic representation/processing, contributes to word representation and processing, particularly for abstract concepts. The Concreteness Effect: Dual-Coding Theory and the Context Availability Model It has been demonstrated repeatedly, and with a variety of methodologies, that concrete words have a cognitive advantage over abstract words—an advantage, labelled the ‘concreteness effect’. With respect to lexical processing, early demonstrations of a processing advantage for concrete over abstract words were provided by James (1975), Whaley (1978), and Rubin (1980). James showed that at least when low frequency words are considered, concrete words are identified as words faster than abstract words. Whaley (1978) and Rubin (1980) adopted a correlational approach, showing that there is a significant negative correlation between concreteness ratings and lexical decision reaction times for the same items. This processing advantage has since then been replicated in both lexical decision (Binder et al., 2005; Bleasdale, 1987; de Groot, 1989; Howell & Bryden, 1987; Kroll & Merves, 1986; Schwanenflugel, Harnishfeger & Stowe, 1988; Schwanenflugel & Stowe, 1989) and word naming tasks (de Groot, 1989; Schwanenflugel and Stowe, 1989). With respect to memory for concrete and abstract words, it has been again repeatedly demonstrated that concrete words have an advantage over abstract words in both long-term and short-term memory tasks (e.g. paired-associate learning (Paivio, Yuille, & Smythe, 1966); serial recall (Allen and Hulme, 2006; Romani, McAlpine, & Martin, 2007; Walker & Hulme, 1999), free recall (Romani, Embodied abstract semantics 5 McAlpine, & Martin, 2007; ter Doest & Semin, 2005), reconstruction of order (Neath, 1997); and recognition memory (Fliessbach, Weis, Klaver, Elger & Weber, 2006)). Among the handful of proposals that have been put forward to explain the ‘concreteness effect’, two have been particularly influential: dual coding theory (Paivio, 1971; 1986; 1991; 2007) and the context availability model (Schwanenflugel and Shoben, 1983; Schwanenflugel, 1991). In both of these accounts, concrete word representations are assumed to be richer than abstract word representations (see also Plaut & Shallice, 1993). According to dual coding theory, concrete words are represented in two representationally distinct but functionally related systems: a verbal, linguistic system and a non-verbal, imagistic system. Abstract concepts, on the other hand, are primarily or exclusively represented in the verbal system. The cognitive advantage for words referring to concrete concepts is attributed to the fact that they have access to information from multiple systems. According to the context availability model, both concrete and abstract concepts are represented in a single verbal code and neither the representations nor the processes that operate on these representations differ for the two types of concepts. The argument here is that comprehension relies on verbal context (either supplied by the discourse or by the comprehender’s own semantic memory) in order to be effective. Accessing the meaning of a word involves accessing a network of associated semantic information and the advantage for concrete words arises because they have stronger and denser associations to contextual knowledge than abstract words. These two proposals have guided research on concrete/abstract semantics; results, however, have been inconclusive. The majority of recent work is neuroscientific in nature, employing either electrophysiological or neuroimaging techniques in order to determine the neural bases of the distinction between concrete and abstract words. Embodied abstract semantics 6 A series of studies using event-related potentials (ERPs) suggested combining dual-coding theory and the context availability model in explaining the concreteness effect (“context-extended dual coding theory”— Holcomb et al., 1999; West & Holcomb, 2000). ERP studies have identified two components associated with concreteness: the N400 and a late negative component peaking around 700-800 milliseconds. With respect to the first component, all relevant studies have found that concrete words elicit a larger N400 than abstract words (Holcomb et al., 1999; Kanske & Kotz, 2007; Kounios & Holcomb, 1994; Nittono et al., 2002; West & Holcomb, 2000; van Schie et al., 2005). The observation that the effect has an anterior maximum but is widely distributed across the scalp (West & Holcomb, 2000) and the failure to find any structural overlap between concreteness and visual object working memory on that component (van Schie et al., 2005) has led to the suggestion that the effect arises within a verbal semantic system that is common to both concrete and abstract words. This N400 component has been argued to reflect postlexical processing in a semantic memory system, possibly involving the integration of semantic information into higher level representations (Osterhout & Holcomb, 1995). According to the context availability model, concrete words are assumed to have stronger and denser interconnections with other concepts in semantic memory than abstract words (Schwanenflugel & Shoben, 1983). In the EEG literature, concrete words are said to activate the semantic network more extensively than abstract words and this extensive activation is reflected in an amplified N400 for concrete words. The second, later, component is assumed to reflect the contribution of mental imagery for concrete words: it is more sustained over time, peaking at around 700 or 800 ms post-stimulus. It is said to be associated with the retrieval of mental imagery associated with concrete words and thus consistent with dual coding theory. Although the imagery-related component is consistent with dual-coding claims that imagery Embodied abstract semantics 7 has a late effect in processing, the greater N400 amplitude for concrete words is harder to reconcile with context availability claims. The stronger interconnections in semantic memory for concrete words according to the model lead to facilitated integration of information. The increased N400 amplitude for concrete words, however, has been interpreted as indexing difficulty in integrating appropriate information (see Kutas, Van Petten & Kluender, 2006 for a review). So the extent to which EEG data actually support the context-extended dual coding theory is questionable. A case for a qualitative difference between concrete and abstract word meanings, thus compatible with dual-coding views, comes from neuropsychological studies where a double dissociation between concrete and abstract words has been observed. Although cases where concrete words are better preserved in the damaged/aging brain are the most frequently reported (e.g., Coltheart et al., 1980; Franklin et al., 1995; Katz & Goodglass, 1990; Martin & Saffran, 1992; Roeltgen et al., 1983; Warrington, 1975), there are cases reporting better performance on abstract over concrete words (e.g., Breedin et al., 1994; Cipolotti and Warrington, 1995; Marshall et al, 1996; Papagno et al., 2007; Reilly et al., 2007; Sirigu, Duhamel, & Poncet, 1991; Warrington, 1975; Warrington & Shallice, 1984). In the imaging literature, although abstract word processing seems to involve activations in a more distributed network of brain regions than concrete word processing (Pexman et al., 2007), there is converging evidence that abstract word processing is associated with higher activation in left hemispheric areas that are known to be involved in semantic processing, e.g, the left inferior frontal gyrus (LIFG) (Perani et al., 1999; Jessen et al., 2000; Fiebach and Friederici, 2003; Noppeney and Price, 2004; Binder et al., 2005) and the superior temporo-lateral cortex (Mellet et al., 1998; Kiehl et Embodied abstract semantics 8 al., 1999; Wise et al., 2000; Binder et al., 2005; Binder et al, 2009). With respect to greater activation for abstract over concrete words in the left inferior frontal gyrus, this finding has been interpreted as indicating more effortful retrieval of semantic information for abstract words, a finding that has been interpreted in some studies as consistent with context availability predictions. Again, however, the majority of the studies use items matched on frequency, but not on familiarity or other relevant variables. For instance, in one of the otherwise best-controlled studies in the imaging literature (Binder et al., 2005), although items were matched on frequency, we found that concrete words were significantly more familiar than abstract words (average familiarity ratings of 534 vs. 471 respectively, t(98)=3.956, p<.001). It may well be that such differences in familiarity underlie some of the effects reported in the neuroimaging literature. When concrete words are compared to abstract words, results have been extremely variable. Although some studies have found activations of left hemispheric regions associated with higher levels of visual processing such as the left fusiform gyrus (D’Esposito et al., 1997; Mellet et al., 1998; Fiebach & Friederici, 2003; Sabsevitz et al. 2005), consistent with the dual coding prediction that concrete word meanings activate relevant imagistic information, a number of studies have failed to find any regions at all that are activated more during concrete word processing (Grossman et al., 2000; Friederici et al., 2000; Kiehl et al., 1999; Krause et al., 1999; Noppeney and Price, 2004; Perani et al., 1999; Pexman et al., 2007; Tyler et al., 2001). Some studies have found more bilateral activations during concrete word processing (Binder et al., 2005; Sabsevitz et al., 2005), while other studies have shown that there is no right hemisphere involvement in the processing of concrete words, and if anything, there are more right-hemispheric activations for abstract rather than concrete words (see Fiebach and Friederici, 2003 for a review). One of the reasons for the lack of consistency Embodied abstract semantics 9 in the results may be that the concrete words used within and across studies differ in terms of their featural composition, which quite reasonably leads to activation of different brain networks in different studies or to lack of consistent areas of activation within the same study. Thus, just as the behavioral and EEG evidence reviewed above, imaging studies do not provide clear support for either dual-coding or context availability calling for new theoretical directions and further empirical investigation. The Concreteness Effect: Testing Dual-Coding Theory and the Context-Availability Hypothesis In the literature, it is invariably assumed that the psycholinguistic constructs of concreteness and imageability tap into the same underlying theoretical construct, i.e., the ontological distinction between concrete, spatiotemporally-bound concepts and abstract, non-spatiotemporally bound concepts. After all, when nothing else is taken into account, imageability ratings explain more than 72% of the variance in concreteness ratings, and up to now the variance that is not explained by imageability has been considered to be pure noise, due perhaps to the imprecise nature of subjective norms. This general assumption is illustrated in the following quote: “Although imageability and concreteness are technically different psycholinguistic constructs, the correlation between these variables is so strong that many authors use the terms interchangeably. Here we make the same assumption of synonymy between imageability and concreteness in terms of theory (i.e., concreteness effects = imageability effects).” (Reilly and Kean, 2007: 158). In fact, concreteness and imageability ratings have been used interchangeably in most of the recent literature in the field (e.g. Binder et al., 2005; Richardson, 2003; Fliessbach et al., 2006; Giesbrecht et al., 2004). However, concreteness and imageability tap into, at least partially, different aspects of semantic representations if native speaker intuitions about them are taken seriously: our analyses of ratings for more than Embodied abstract semantics 10 4,000 words in the MRC Psycholinguistic Database show that the frequency distribution of concreteness ratings is bimodal, with two distinct modes for abstract and concrete words (see also Cartwright & Nickerson, 1979; Nelson & Schreiber, 1992), while the distribution of imageability ratings is unimodal (Figure 1). In other words, concreteness ratings capture the categorical ontological distinction between concrete and abstract words (and their underlying conceptual representations), while imageability ratings index a graded property that is meant to capture the differential association of words with sensory (primarily visual) properties. Moreover, from a theoretical point of view, imageability ratings are a proxy for concreteness only in the dual-coding theory, not, for example, in the Context Availability hypothesis because only the former explains differences between concrete and abstract words in terms of whether (and to what extent) the non verbal imagistic system is engaged (Reily and Kean, 2007; Fliessbach, Weis, Klaver, Elger, & Weber, 2006). According to the Context Availability hypothesis, however, imageability would not exhaust the differences between concrete and abstract words which, instead, arise as a consequence of different degrees of richness of semantic representation within a verbal system. Figure 1. Density plots for concreteness and imageability ratings for 4,274 words from the MRC Psycholinguistic Database. Using the dip test (Hartigan & Hartigan, 1985) we rejected the hypothesis of unimodality for the concreteness distribution (dip = .0244, p<. 001) but not for the imageability distribution (dip = .0058, n.s.). Embodied abstract semantics 11 One approach to test both hypotheses is to manipulate concreteness while controlling for both imageability and context availability. Both dual-coding and context availability theories predict that concreteness effects will not be observed under these conditions. Experiment 1 In this experiment, we contrast morphologically simple abstract and concrete words that have been matched for imageability and context availability (as well as a host of other noise variables). Method Participants. Fifty-eight native English speakers (32 female; mean age: 28.69 ± 9.96) participated and were paid at a rate of £6 per hour. Three participants were replaced because of a high number of timed-out responses in their data. Embodied abstract semantics 12 Materials and design. Forty concrete and 40 abstract monomorphemic words were selected (the full item list appears in Appendix I). The items differed on concreteness, but were matched pairwise on 12 lexical and sublexical variables, including rated context availability (see Table 1). Imageability, familiarity, and age of acquisition ratings were obtained from the MRC Psycholinguistic Database (Coltheart, 1981). Items were also matched in length (in number of letters, phonemes, and syllables) and number of meanings (in terms of number of synsets in which a word appears in WordNet; Fellbaum, 1998). Frequency, orthographic neighbourhood density, mean frequency of orthographic neighbours, and mean positional bigram frequency were taken from the English Lexicon Project (ELP; Balota et al., 2007). Finally, we obtained context availability ratings by asking 47 native English speakers to rate words on a 7-point Likert scale according to how easy it is to come up with a particular context or circumstance in which they might appear. The instructions to participants were identical to those used by Schwanenflugel et al. (1989), with the exception of some of the examples given in order to anchor the ratings which differed between the studies. We obtained these norms for 650 words (each word was rated by 22 or 25 speakers). See Table 1 for details and Appendix I for a full list of items used in the experiment. We also selected 40 concrete and 40 abstract words matched with the experimental items in terms of concreteness to serve as the basis for creating the pseudowords for the experiment. We created pseudowords by altering a single letter in one of these words. We made an effort to select pseudowords with only one orthographic neighbour (the intended real word). In cases in which that was not possible (for all 3-letter and some of the 4-letter words), the intended word was the most frequent among the set of orthographic neighbours of the nonword. The resulting pseudowords were Embodied abstract semantics 13 matched pairwise with the experimental items in terms of length and mean positional bigram frequency. Table 1. Item information for Experiment 1 (averages and standard deviations). The numbers reported here are based on 38 items per condition (two were excluded on the basis of low accuracy; see Results). Abstract Concrete Concreteness 345 (40) 552 (44) Context availability 568 (46) 566 (52) Imageability 500 (42) 505 (35) Familiarity 504 (70) 505 (67) Age of acquisition 385 (40) 390 (103) Log frequency 9.02 (1.44) 9.03 (1.62) Number of letters 5.55 (1.20) 5.63 (1.28) Number of phonemes 4.71 (1.33) 4.55 (1.27) Number of syllables 1.68 (0.57) 1.68 (0.70) Mean positional bigram freq. 1491 (959) 1595 (943) Num. ortho. neighbors 2.63 (3.90) 2.84 (4.02) Mean neighbour freq. 4.86 (3.93) 4.26 (4.13) Number of synsets 5.16 (3.25) 6.50 (6.86)Concrete Concreteness 345 (40) 552 (44) Context availability 568 (46) 566 (52) Imageability 500 (42) 505 (35) Familiarity 504 (70) 505 (67) Age of acquisition 385 (40) 390 (103) Log frequency 9.02 (1.44) 9.03 (1.62) Number of letters 5.55 (1.20) 5.63 (1.28) Number of phonemes 4.71 (1.33) 4.55 (1.27) Number of syllables 1.68 (0.57) 1.68 (0.70) Mean positional bigram freq. 1491 (959) 1595 (943) Num. ortho. neighbors 2.63 (3.90) 2.84 (4.02) Mean neighbour freq. 4.86 (3.93) 4.26 (4.13) Number of synsets 5.16 (3.25) 6.50 (6.86) Procedure. Participants were tested individually. Each trial began with a fixation cross presented in the middle of the screen for 400 milliseconds, followed by presentation of the string for 2000 milliseconds or until a response was given (whichever was earlier). Participants were instructed to respond as quickly and accurately as possible using a serial response box. After response or timeout, the screen went blank and participants were instructed to press the space bar to continue with the next trial. Ten practice items were first presented, followed by the 320 words and nonwords presented in a different random order for each participant.